Do the Hard Stuff First: Scheduling Dependent Computations in Data-Analytics Clusters

We present DagPS, a scheduler that improves cluster utilization and job completion times by packing tasks with multi-resource requirements and inter-dependencies. While the underlying scheduling problem is intractable in general, DagPS is nearly optimal on the job DAGs that appear in production clusters at a large enterprise. Our key insight is that carefully handling the long-running tasks and those with tough-to-pack resource requirements will lead to good schedules for DAGs. However, which subset of tasks to treat carefully is a priori unclear. DagPS oòers a novel search procedure that evaluates various possibilities and outputs a valid schedule. An online component enforces the schedules desired by the various jobs running on the cluster. In addition, it packs tasks and, for any desired fairness scheme, guarantees bounded unfairness. We evaluate DagPS on a 200 server cluster using traces of over 20,000 DAGs collected from a large production cluster. Relative to the state-of-the art schedulers, DagPS speeds up half of the jobs by over 30%.